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>As assessment developers, it is our responsibility to 
ensure that the information displayed on score 
reports are understood, meaningful, and useful to 
their intended audiences. 

■ Standards for Educational and Psychological Testing 

■ Code of Fair Testing Practices in Education 

■ Code of Professional Responsibilities in Educational 
Measurement 

>We contend that the efficacy and utility of reports by 
consumers is an important facet of validity. 

>We also contend that the delivery of good score 
reports is an issue of fairness. 


2 



inspiring minds'" 


Our Approach 


More recently, the College Board Introduced more rigorous 
methods in the development of score reports in the last ten years. 

Overview of the Approach: 

1 . Considering the purpose of the test, clearly articulate the audience and goals of the 
score report. 

2. Survey consumers using rigorous designs to capture needs and preferences 

3. Select and utilize appropriate psychometric methods 

4. Develop a mock-up of the score report 

• Keep true to the purpose 

• Provide context 

• Give guidance 

• Keep it simple 

5. Capture feedback of mock-ups 

• Strive to use representative samples 

• Ask preferences and comprehension-type questions 

6. Revise & pilot 
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Several sets of activities/efforts changed the 
College Board’s approach 

Many good and bad things were learned when we actually asked 
educators what they thought and how they used score reports. 
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SAT® Skills Insight™ Development 


The College Board engaged many external educators, curriculum experts and 

users to develop and validate the content in SAT Skills Insight... 

Step I: Agree on the purpose of the score report 

• Provide enhanced feedback to students and educators 

• Give them insight into what the score represents 

Step II: Conduct Scale Anchoring Research 

• Ronald Hambleton, Stephen Sireci (University of Massachusetts-Amherst), 
and Mary Lyn Borque (formally with the National Assessment Governing 
Board) implemented the scale anchoring approach to the SAT. 

• Using secondary and post-secondary educators and content experts, they 
developed scale anchors at 200-290; 300-390; 400-490; 500-590; 600-690; 
700-800 for critical reading, mathematics, and writing. 

• Test Development staff with R&D convened several panels for each section 
of the SAT to develop performance category descriptions that were accurate 
and understandable by consumers. 
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SAT Skills Insight Development 


Step III: Test Concept at Focus Groups 

• 2007 National Forum focus group with students 

• Round I of focus groups with students and counselors 

• Round II of focus groups with students 

Step IV: Validate and Iterate on the Content 

• Secondary and post-secondary educators 

• State and district curriculum and assessment experts 

Step V: Conduct Usability Testing 

• Usability studies with students and teachers 

Step VI: Test Concept at More Focus Groups 

• Focus groups with state educators to develop aggregate reports 
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Round I of Focus Groups 

with Students and Counselors 

Focused on the following: 

> Feedback and initial reactions to skill descriptors, 
suggestions for improvement, and sample 
questions 

>Gauge intended use 

> Feedback on reactions to report format, language 
and overall layout 
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Round I of Focus Groups: 

Key Findings 

> Initial reactions to report are mixed among students and counselors 

> At first glance, the information seems overwhelming and challenging to interpret. However, both 
audiences see that it is a tool for improving skills 

> The availability of information on all score bands was of great interest 

> Skill descriptors contain too much educational jargon. Needs to be more 
“student friendly” 

> Skill Improvement section could be improved by identifying current 
weaknesses in skill areas. 

> Examples of SAT® Questions should be provided for ALL skill descriptors 

> Needs an eye-catching format (adding colors could enhance appeal of report) 

> This tool would be beneficial if provided after the PSAT/NMSQT ® and prior to 
the first SAT 
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Round II of Focus Groups: 

Key Findings 

> After making revisions from the first round of 
focus groups, the students indicated the following: 

■ They could easily navigate through the website. 

■ The layout of the webpage was appealing. 

■ The most important feature of the report was the 
inclusion of the sample items. 
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SAT Skills Insight (final version online): 
Student Version 



How does Skills Insight work? 




"Whot Is II?" 

SAT Skills Insight Is a free tool to help you get a score that really shows 
your abilities. It identifies the types of skills that are tested on the SAT 
with sample test questions so that you can do your best if you’re taking 
the SAT for the first time, or if you got your SAT scores and are thinking, 
'‘I want to do better than that,'' 

Keep In mind thati 

• The SAT shows how ready you are for college-level work, 

• If you improve the skills needed for college, you can improve your 
SAT scores. 

• You are likely to do better on the SAT if you're more familiar with 
Its content, 

Taking the SAT shows colleges that you want to achieve more In life, SAT 
Skills Insight helps you figure out how to get where you want to go, 



^5^ Done 
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Students receiving their 
score reports on the web 
(via My Online Score 
Report) are provided with 
a link to SAT Skills 
Insight. For those 
receiving their score 
reports on paper (in the 
mail) the uri to skills 
insight is printed on their 
score report. 

www.collegeboard.com/satskillsinsight 
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OVERVIEW 


MATHEMATICS 


WRITING 


SAT® Skills Insiaht^ 


300-390 


500-590 600 - 69 [ 


Using 5AT(^ Skills Insiaht^^ O 

CRITICAL READING SKILL GROUPS: 1 2 3 4 5 

1. Determining the Meaning of Words 


Academic Skills 


A typical student in this score band can dc the 
follcwing: 

• SKILL 1: Use context clues when selecting 
missing vocabulary at the sentence level 

• SKILL 2: Use knowledge of root wordsj prefixes^ 
and suffixes when selecting missing vocabulary at 
the sentence level 

SKILL 3: Use the context of a sentence or a short 
section of text to clarify the meaning of unknown 
words or to select the appropriate meaning of 
familiar and simple words that have multiple 
meanings 


Suggestions for Improvement 


To prepare for the next score bandj try the 
following; 

When you come across an unknown or difficult 
word in your reading^ look at the word to see if 
it contains a root word that you know. 

When you come across an unknown or difficult 
word in your reading^ try to find out if that word 
is jargon^ or the specialized vocabulary of a 
specific field. 

^ When you come across an unknown word or a 
word with multiple meanings in your readingj 
look at the context of the sentence for clues to 
what the word means. 

^ In your readingj pick out a long sentence and 
break it down into smaller parts. Think about 
how the structure of the sentence creates 
relationships among the ideas in the sentence. 
Think about how words like but, although, and 
also create certain relationships. 


SAT Skills Insight - Student Version 


Skill Examples 


The example questions below demonstrate the Academic Skills found in this score band. Without looking at the 
answersj try out the questions to see how comfortable you feel with the skills they test. 


SKILL 1-EXAMPLE 

Use context clues when selecting missing vocabulary at the sentence level 


Some fans feel that sports events are only when the 

competitors are of equal abilitVj making the outcome of the 


A successful . . assured 
B boring , , questionable 
£ dull . . foreseen 
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step VI: Refine report for state officials 

> Using the feedback that we received from students, 
teachers, counselors, and other educators, developed 
an aggregate report for use by state officials. 

>ln addition to providing the performance category 
descriptions, sample items, and suggestions for 
activities to enhance performance, provided a 
summary of student performance across the state: 

> Showed overall performance over time 

> Offer comparisons 

> Show patterns of performance 
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Skills Insight Paper Report for State Officials: 
Original Designs 




SAT SKILLS MAP - 2007 STATE SUMMARY 


Overview ol Student PerTormance on Uie SAT 

STEP t Exemlne tHe cbarte Eielowta gauge tHe perlormanoe ulyaui stuflenta over me lastmree yeera. 
Keep In mind that; 

^#1 Students, can Improi-ie lhe^r score by learning about end pracadng the t>-pes oT quasdons Ihal are on the SAT. 

^4 The SAT Stlls Map Is one or 5e\,^ral tods trial can assist you In helping your students succeed. 


CRITICAL READING 




Ilig '^Ical 4ig tc« 4ig nulille euh '^Uig dUnDiiilkn. 



STEP 25 Examine me charta below to develop an understanding dI bow your studenta perTormed on me SAT by acore band, 

^4 In tre Piiflap, tte've gcuped Ihe SAT scores Into sh ■’score bands ’ bel'^'een 200 and 910. Fit es: li band, tire Pi^lap pro'Ales examples- at a 

typical student's sHIts iiXtio asiiresvirittilnlhat tend '[See pagss2 -7 )l TTe InTonnallon In the Map Is tiasedon the perTomnanceor stucbntsvrfnD 
ha-nelahen IheSAT. IREFERENCE SCALE ANCHOR RESEA'F=iCHI 


■ apgcd 'iikiitf lar agritfdl 
dMHiSiloog srudi ucra ag'c 


MHfi 

hni^i. 

Hrtm 
II- IHHT 

atHt-aao- 


H- a*l73i 
4 <KM 40 ' 


4 ^ l«« 


• apg£d ’cbA' 3 Ai>tniil 
dACEHjUlonft !ir uUi bxra !U'c 


1 CH 


• apgLd 'elsg'^raAiitigl 

sT auIi ucra S 6 ’S 


lyk -iCH 


• SpgLiti jgiwdl 

duunf^dont sT uili ucra aa'a 


T? 17 ^ 


• Spgcg agrit^gl 

dunfpUont sT udi ucra qa'c 


^5 HIT 


• apgcg 'cbA'^raAnKgl 
CAiHlpikm !f udi ui!ra !U*C 



-• apuA h£kb' OKI Aral 
dHCflpUem^ UL* uoia ognc 


- apud IkUa' atntfal 
dAudpiiem^ U 6 s uxn u 


Hilkin 

H- 2 ar.iiT J 
BOMW 1 * 



” apard haklA' 0 tnAr.iJ 
dauflptlem ei»ia mk: 


- apuA h£klA' atfiAral 
dAUilplieriftiiii aki u»ta hiiu 


h- 3 m;ih J 

3W«W 15 lOT 


- apuA hciklA- owiAral 

dAicflptiem s^ ak* uoia agnc 




- apA£A hcklA- otnAr,iJ 
dAUflptiem^ A*££ uoA !mh: 




7 D 0 JM i'l 


34 : . 4 ] • 

AM- 4 M IT" 


1 5 pK* iKUArtir aAflAiil 
dAur4^a£& ii AKh uora band 


T 5i:Kiii holdAr lor aAna'U 
dAfi£r.|^a!A n AHh uora band 


S^K:* holdAr lor aAnanl 
dAE£r irf AKh uota baiKl 



• BpK* holdAr tir gAflwil 
dAUfi^ava iij AKh uora band 


T Si:Kii holdAr lor aHiLil 
dAS£f.(^&!& (A AKh KOTA band 


> 3j For eacli section oftbe SAT - cntlceil Heading, Matheniatlca^ and Writing - examine me typea oI aHlLLa tested In eBcH score 
band and me suggestions for Hupnovlngta me next level to nnderatandwbat your atudenis sfitwaTtf Know to succaad., and 
bow they can improve tbelr abllla. 


O 


m 



Boxplots 
showing mean 
and range of 
scores of 
middle 50* 
percentile 



Percent of students 
(State & Nation) 
falling within each 
score band on each 
section of the SAT. 
General Descriptions 
of the skills students 
have mastered at 
each score band were 
going to be provided 
in the space-holders 
next to each display. 


a 


inspiring minds" 




Skills Insight Paper Report for State Officials: 
Original Designs 


Atypical atudont in this soore interval can do the following: 
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Skills Insight Paper Report for State Officials: 
Revised Designs 


a 


OoUegeBoard SAT 

'comiuect bo ooUsgs suDoesB* 


SAT Skills Insight - 2008 Stats Summary 

Sample Stale » Public Sahoole 


St^p 1: ExamlQ# th& charts b^low to gaugo tb>9 poirformancoi of your studoat-s on oach soctlon of tho SAT ovor tho last fivo yoars. 

Kq^ in ETjind that participation rates for your state may vary from year to year. 

Sample State - Public Schools Participation Rate* 


Numt-ei of Test Takers 


2004 

2005 

2006 

2007 

2008 

47^ 

4S% 

45% 

43% 

42% 

3.367 

3.330 

3.514 

3.449 

3.341 


The charts below display scores for your higher perfomiing students [7Bth percentile) and lowrer performing students (2&t±i 
percentile), as well as. mean scores for all of your students, over t±ie last five years. 

Critical Reading Trends 

— # — Students scoring at the 75th percentile — ■ — Moan A Students scoring at the 25th percentile 


■ = — — 1 1 ^ - H 2 &21 

A ^ ™ — ■ “ — ■ ■ A 4 50 


eoo n 


700 - 


600 - 

600 # 

500 - 

520 □ 


450 A - 

400 - 


3m - 


200 - 



2004 


2005 


2006 


2007 


2006 
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Skills Insight Paper Report for State Officials: 

Revised Designs 



Step 3: Examine the charts l>elow to develop an understanding of bow your 2006 College-Eksund Seniors performed on the 
SAT by score band. 

^ Th 0 charts below display tiiQ percentage of students vdthin your state, and the Total Group, whose scores fe]] within each of the 
six score bands for CriticaJ Reading, Mathematics, and Writing. 


□ Sample State 

Public Schools 

□ Total Group ■ 

■ Public Schools 



Critical Reading 


Mathematics 


Writing 

200-290 

J 3.0% 

209-290 

1 1.0% 
J 2.5% 

200-290 

o.a% 

J 2.3% 

30(^390 

1 5.9% 

30(^390 

|7.0% 

30(K390 

1 6.7% 

1 14.7% 

1 13.4% 

1 16.8% 





- 


400490 

1 18.4% 

400490 

1 20.0% 

400490 

1 21.0% 

1 32.9% 

1 29.8% 

1 34.9% 





- 


500-590 

1 34.1% 

500-590 

1 34.0% 

500-590 

1 30.4% 

1 29.9% 

1 31 .2% 

[mo% 





- 


0OM9O 

1 29.2% 

50M90 

1 26.8% 

60M90 

1 26.1% 

1 15.4% 

1 17.7% 

1 12.9% 

70M00 

1 11.7% 
^ 4.0% 

70M00 

1 10.7% 

1 5.5% 

70(0300 

1 9.0% 

J 3.6% 

Step 4] Review the academic skills tested within each score band of the SAT and the suggestions for improving to the next 
higher scoreband to understand the ty[>es of skills your students should master to be successful in college. 
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Skills Insight Paper Report for State Officials: 
Revised Designs 

SAT SKILLS INSIGHT 

Mathematics 

Acadeinic Skills at Each Score Band and Suggestions for Improvement 



200-290 

300-390 

400-490 

NUMBER AND OPERATIONS 

Acadeinic 

Skill** 

Ab^|[ 3 laludenblnlbia 

aoaEirdErvElcandD 

DiElDinwIig; 

■< kJenOrvlacbaabdwhDle 
nuntiora 

« 8 d^e wad problems uahg 
aldtllbn aubdrallba 
muDIpIcadlob.arddMalDn 
od wtnie nunAera 

« HE[ 3 llb 33 l[m 3 DiEmablcai rada.-'dElnlOnnaabDiJle^nBnblai 
nodadltfx hcludrig aclErdincndbadlon 
« lden 0 lY 3 nJe 1 baddeaadbeaanurneflc 3 lp 3 blEmlnaaEquen[E 

« UenDry, uaa, and rEpraaEnI IrEcllona and perccrda t\ 3 dDmiEll[ and 
aigdTEt aeldnga 

« Uae propBfblea nr bveh and odd numbers, miJbplEa, and -^bora 
« UendrYand uaeDieneniearorplacE^uoalnaDhrigprEUerTialnwMng 
dedmai repreaanl^bldna |e.g., taiDia 3 rd brndredDiaj 
<■ Uae propablea nr hequailDea Di cbmpEre and Drder rumbera 

Suggestioitsfor 
Improve iient 

Tbalvan[ElDlhene«b 
hIgbEab aoaE band, 
alucbrdaabDUdlncua 
cnDiElDlowligatll^; 

« Hecall baat matienEllcai 
1 ac 1 a,demt 1 loflaabaj 1 
eigitnrdlal notadba 
ticimiigaclEndncnDdadldn 
» kJenDrvanJElhad 
deaadbEa a numerlcEl 
pabbem ri 3 ae^jUETioe 

« IdaiblY, ice, and rEprEaEnd IraMlona 3 rd pacErda In 3 d 1 bmEll[ and 
ElgEbrataElbliga 

« L^EpropErlleaDrEVEnTid tddnundera,mJ 1 ^Ea,andl 3 cbdra 
" IdaiDly and uaE Die rianiea ba place vaiuEa In adding pmbiema 
IrwEivlig decimai repreaanbabldna |e.g., benDia 3 rd hirdrEdDiaj 
< L^E propEniea nr InequailDea be campErE and order nunbaa 

« Sd^e problems uang hdeEalrombEaJc aebDiEDrY 3 rd baalc number DiEDrY 
< HEbd^zE 3 rd ^lY radk\ prcporDm DrpercEnd In aDMng pmbienia 
« UaepropableaDrrEElniinbEroperEDDna.ordedng.ardDie bero-pmdud 
priperbY 

« SDhE problems- rwEivngcDmbnglEdriiqijea 


500-590 



« SalK prctiem^ using Ide^t rnm ^ Iheuryaid bBUi nimber 
thenrY 

« Riecugnbe aid ^pply rsIlD, prapDitkin, or ptnent t\ ^Mng pniblefnE- 
" U£c propenitG or reol nurrdor cpo rad Ions, DidEf hg, aid Ihe ztiD- 
productpmpatY 

< ^IwprctiEm^lnwMngcouMIngtschnlqutt 


600-690 


700-800 


« Deltnnlie vsIue^ a properlle^ or nunber £■ t\ a ^UEnoe glwn « U^n t\ sigsbralc ond gEantlrt oonbesbb 
3 dEEcrIpllcn M Itit ^guaico ■ crosts aid u £0 raOn^, IracOnni, nr pacont^, Includlr^g 

■ Cresde Tti \st rsdlci^, rractkinE, a pEroails li EJi^Ing pmblem^ sigEbraic Eiqiroa^lofia, ti aoHng problana 

" ^1 w moiE-CHTiplEX oounOng problems |e.g., pbrnuilsdlcrF^, 

[oniUnsIlErFb, snd ricli£Joa^Kli£Jon| 


< DPlEimnE vshJEaa propErllts or nmbtr^ In s ^equEnoe ^n glwn « Use t In olgebrolc and igonnKtdc [nnlEsda 
3 daacdpllm d1 Bie K^UEfice « cneale Tti ihe rartlM. rractlona^ a pEroenta, Includng algebraic 

« CTE^E3ncluaEra1lDa,1r3[1lona, DrporcETdariaaMng problems Eiqiraaalofla, rianMng problems 

« ^l¥e mnro-cnmfilEic CDirbng problEma |e.g., pEnnulEODna, 

[OfTtinEOcra, 3 rd Inclmlorv^lu^lonl 


Tbia H Ihe bop acnrs tiand 3rd aludEMa vfa acore ad 1bla 
level will bavE lltElYmaMEf Ed Die atllla llalEd 3t ail odher 
levela. Howeva, aludErdacanalwaYationElllromrnaE 
precllce. W'i encouragE alidanta bo review tpe atllla-and 
ei^rrpiEa llabed In 1hE 5DD-59D and 600-6^0 acore bands. 
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Skills Insight Paper Report for State Officials: 
Revised Designs 


Score Band 200-290 


Number and Operations 

Skill 1: Identify factors of whole numbers 


Each of the following is a factor of 80 EXCEPT 

(A) 5 

(B) 8 

(C) 12 

(D) 16 

(E) 40 

Ainswer: C 
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Lessons Learned 


> Throughout the development process, we needed to have razor sharp focus 
on the purpose 

• It sometimes became easy to drift from the original purpose (and we did) or add more 
than what was originally planned. 

> Must use empirical (scientific) methods to capture feedback 

• Don’t assume... 

• Be careful of sampling issues 

Don’t just ask your friends! 

> The manner in which questions are asked will influence the responses you 
get. 

■ As Ron Hambleton and others have recommended, ask comprehension-type questions in 
addition to preference. 

> Once delivered, need to assess how score reports are used. 

> Need ongoing feedback loop to inform score report enhancements and 
improvements. 
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Immediate Next Steps 


> Examine the validity of the information provides 
from multiple perspectives: 

■ Examine the use of the SAT Skills Insight score reports 
by state officials. 

■ Examine trends over time of student performance at 
each score anchor taking into account their curricular 
experience. 

■ Examine whether interpretation is similar across 
different types of consumers. 
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Vision 


The College Board is focused on the integrated use of tests in helping 
educators make decisions about students. 


College Board Standards for College Success 


Secordary 


Slate Standards 


Ptet'Secondary 


Steps Toward CoUege Readiness Coliega Readiness 


Coftege Outcomes 
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Next Steps 


The College Board is transforming its score reports building on 
current efforts to provide more comprehensive feedback to students 
and educators about examinees’ readiness for coiiege 




The purpose of score reports is changing from 
performance on one of the College Board 
assessments to feedback on college readiness. 


College 

Outcomes 



Goals of Report : 

1 . Provide the scores 

2. Compare to others 

3. Indicate stability of scores 

4. Show growth 

5. Offer meaning of the scores 




23 


inspiring minds" 



Questions, Comments, Suggestions 

• Researchers are encouraged to freely express 
their professional judgment. Therefore, points of 
view or opinions stated in College Board 
presentations do not necessarily represent official 
College Board position or policy. 

• Please forward any questions, comments, and 
suggestions to: Thanos Patelis 
tpatelis@colleaeboard.org or 21 2-649-8435. 

Thank you!! 
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